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ABSTRACT 



A cognitive demands analysis of a learning technology, a 
term that includes the hardware and the computer software products that form 
learning environments, attempts to describe the types of cognitive learning 
expected of the individual by the technology. This paper explores the context 
of cognitive learning, suggesting five families of cognitive learning. These 
families are: (1) content understanding; (2) collaboration; (3) 

communication; (4) problem solving; and (5) metacognition. Three technologies 
are analyzed with reference to these types of learning. The first technology, 
Algebra Tutor, is an intelligent tutoring system to help students learn 
algebra skills. "Hands-On Universe" is an astronomical research environment 
for high school students. "Function Machines" is an application for a visual 
programming language designed for mathematics and science education. Analyses 
of the types of learning expected for each of these technologies have 
resulted in the development of a suite of performance tasks (an integrated 
simulation) that includes individual and collaborative concept mapping tasks, 
a problem-solving search task, an explanation task, and a metacognitive 
questionnaire. This assessment environment is integrated across grade levels 
and within content areas to blend real-world tasks with a project-based 
scenario that captures many types of cognitive learning. (Contains 1 table 
and 79 references.) (SLD) 
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The Five Families of Cognitive Learning: 

A Context in which to Conduct Cognitive Demands Analyses 
of Innovative Technologies 

The purpose of this paper is two-fold: 1) provide an intellectual context of this type of 
learning that my colleagues today are assessing. We call this context the five families of 
cognitive learning; and 2) provide a context for the type of learning demanded by the specific 
technologies that we are assessing. As technology becomes more integrated into general 
classroom instruction and as assessment draws learning closer to instruction, it becomes 
increasingly important to find ways of evaluating what students are learning via these 
technologies. By “technologies” we refer not only to the actual machines, but also to the 
learning environments (e.g., software products), which afford students the opportunity to 
leam. We need to understand what a particular software product does and how students learn 
from using it. Whereas cognitive task analysis attempts to identify the cognitive skills an 
individual uses or needs to perform a task proficiently (Klein, 1995; Means & Gott, 1988; 

Roth & Mumaw, 1995), a cognitive demands analysis seeks to describe the types of cognitive 
learning expected of the individual by the technology. Clearly, this learning is dependent upon 
a number of factors, including both the context in which the technology is used and the 
individual characteristics of the learner. However, within a given learning situation, we can 
attempt to understand what is expected — or demanded — of the learner. 

Information regarding the cognitive demands placed on students using the technology can 
simplify the assessment process by allowing the alignment of student assessments with the 
technology. Students are not expected merely to leam the content being presented to them; 
rather, learning demands may also include the types of activities and instructional opportunities 
in which students are expected to engage. By identifying these types of expected learning, we 
can create a suite of performance assessments integrated into a real-world, problem-solving 
environment that can assess varied student learning and understanding. 
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Context: The Five Families of Learning 

In order to carry out a cognitive demands analysis, we must examine the types of learning 
that a particular technology targets. In the CRESST model of learning, Baker (1995) posits five 
families of cognitive learning: content understanding, collaboration, communication, problem 
solving, and metacognition (see Figure 1). These cognitive types of learning owe their 
intellectual history to Gagne (Gagne, Briggs, & Wager, 1992), Mayer (Mayer & Wittrock, 
1996), Merrill (1983, 1993a, b), and Salas (Salas, Dickinson, Converse, & Tannenbaum, 

1992). The five families describe the range of cognitive learning in which students engage; they 
are seen as working together to influence overall learning. Once we determine the types of 
learning in which students engage, we can create assessments to evaluate each learning family. 




Figure I. The CRESST model of learning. 



Content understanding. The first type of learning is the understanding of subject 
matter content. Whereas the other types of learning involve relatively context-free material — 
that is, learning that could occur within any number of varied contexts — content understanding 
refers to the learning of domain-specific material. For instance, students engaged in learning 
astronomy via the use of an innovative technology will learn particular facts, concepts, 
procedures, and principles related to astronomy. 
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Assessment of content understanding should evaluate not only basic factual knowledge, 
but also a deeper level of understanding of the subject area (Herman, 1992: Linn, Baker, & 
Dunbar, 1991). There are many approaches to this assessment: explanations or essays, 
representational tasks (e.g., concept mapping), multiple-choice questions, and so on. The 
CRESST content understanding model (Baker. Aschbacher, Niemi, & Sato. 1992) uses 
student explanations to assess comprehension: this model has been used successfully for 
assessing deep understanding in history, geography, mathematics, science, and 
interdisciplinary tasks at the elementary, middle school, and high school levels (Aschbacher, 
1995; Baker, 1994; Baker, Aschbacher, et al., 1992; Baker, Niemi, Herl, et al., 1995; Herl, 
Baker. & Niemi. 1996; Niemi. in press). This assessment model includes the following 
activities: stimulating prior content area knowledge, reading primary source documents 
containing new information, and writing an explanation of important issues that integrates new 
concepts with prior knowledge. Understanding is assessed by examining overall content 
quality, prior knowledge, principles, and use of resources. 

Another approach to the assessment of content understanding is to evaluate students’ 
underlying knowledge structures. Research on memory suggests that knowledge is organized 
in complex semantic networks (Jonassen. Beissner. & Yacci, 1993; Rumelhart & Ortony, 
1977). Ausubel (1968) posited a hierarchical memory model in which new concepts are 
integrated hierarchically into existing cognitive structures and relationships generated 
accordingly. Rather than constraining cognitive structures in a hierarchical arrangement, 

Deese s associationist memory model (1965) allowed for various types of cognitive structures. 
Expert-novice research has provided additional information regarding individuals’ knowledge 
structures, indicating that expert knowledge is organized in a qualitatively different way than is 
novice’s knowledge (Chase & Simon. 1973; Chi, Feltovich, & Glaser, 1981; Chi, Glaser, & 
Farr, 1988; Gentner & Stevens, 1983). As novices become more expert, a restructuring of 
their knowledge occurs (Royer, Cisero. & Carlo, 1993). Whatever the exact nature of 
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students’ knowledge structures, by eliciting specific information from students we can attempt 
to assess these cognitive structures. 

A concept map is a graphical representation of information consisting of nodes and links, 
or labeled links. Nodes correspond to concepts within a particular subject area or domain; links 
indicate relationships between pairs of concepts (or nodes), and labels on each link explain 
how two concepts are related (refer to Jonassen et al.. 1993, for more in-depth coverage of 
concept mapping). Students create concept maps by identifying important concepts and 
generating and appropriately labeling the links between those concepts. This approach assumes 
that a deep understanding in a subject domain allows an individual to conceive a rich set of 
interrelationships among important concepts within that domain (Heinze-Fry & Novak. 1990; 
Novak & Gowin, 1984). Asking students to show relationships between important concepts 
by creating these maps, we can evaluate their content understanding (Baker, Niemi, Novak, & 
Herl. 1992; Herl et al., 1996; Jonassen et al., 1993; Ruiz-Primo & Shavelson, 1995). Concept 
mapping relies less on verbal ability and is less dependent upon language skills than an actual 
essay writing task, while still requiring other skills involved in such a task. Thus, using a 
concept mapping approach allows us to separate the assessment of content understanding from 
the assessment of communication skills. 

Collaboration. The second type of learning, collaboration, involves learning how to 
cooperate with members of a team. Students learn to work together, each contributing to the 
group in his or her own way. Collaborative learning in the classroom can foster student 
learning and deeper understanding, as well as higher levels of self-esteem, attitudes towards 
others, and social skills (Webb, 1995; Webb & Palincsar, 1996). In addition, teamwork has 
been studied from a workplace readiness perspective: Interpersonal and teamwork skills are 
now recognized as essential for future job preparedness (O’Neil, Allred, & Baker, in press). 
Researchers in education, industry, and the military all recognize that collaboration can enhance 
learning, task performance, work productivity, and product quality. In a workplace 
environment in which teams are believed to offer the potential for greater competitiveness. 
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employees are increasingly being asked to work in teams. For all these reason, organizations at 
the national, state, and local levels have begun calling for the use of collaborative group work 
in the classroom (for example, California State Department of Education. 1992b: Mathematical 
Sciences Education Board. National Research Council. 1989: National Council of Teachers of 
Mathematics. 1989: O’Neil, in press). 

Teamwork is thought to be composed of two sets of skills: taskwork skills and teamwork 
skills (Morgan. Salas. & Glickman. 1993). Taskwork skills influence how well a team 
performs on a particular task. Teamwork skills influence how effective an individual member 
will be as part of a team. Further defining teamwork skills. O’Neil, Chung, and Brown (in 
press) identify six factors that affect collaboration. Adaptability refers to a team member’s 
ability to recognize problems within the team and respond appropriately. Communication 
involves the clear and accurate exchange of information among team members. Coordination 
refers to how team members organize their activities to complete a task on time. Decision 
making involves using available information to make appropriate decisions for the team. 
Interpersonal skills affect the cooperative interaction of the individuals within a team. Finally, 
leadership is a team member’s ability to provide direction and coordinate activities for the team. 
In combination, skill in these factors determines how well a student collaborates with his or her 
teammates. 

One way to assess students’ collaborative skills is through observation of group work; 
factors affecting collaboration can be thus be evaluated for each team member. Although 
observational methods are generally difficult and time-consuming, evaluation of collaborative 
activity becomes easier through the use of technology. If teammates communicate via 
computer, on-line messages sent back and forth between them can be saved for subsequent 
coding as process measures. Collaborative environments that present a set of constrained 
messages from which to choose (rather than allowing any message to be sent) can further 
simplify the coding process (O’Neil. Chung, & Brown, in press). By examining on-line 
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messages, we can better understand the process by which a collaborative group arrived at its 
final product, as well as the contribution of each member of the group to that final product. 

Communication. A third type of learning involves the communication of ideas. 
Communication is the ability to express oneself clearly and effectively — both orally and 
through writing — for various audiences and purposes. A more team-oriented definition sees 
communication as the process by which information is clearly and accurately exchanged 
between people, often in a prescribed manner using proper terminology (O’Neil. Chung, & 
Brown, in press). Throughout their lives, students will benefit from the ability to convey their 
beliefs. Whether by an oral presentation, through some form of writing, or by using some 
multimode approach, students must learn how to express themselves to others. This expression 
can take many forms, including persuasive, narrative, expository, explanatory, prose, and 
even questioning. 

Much has been documented about the writing process, and scoring rubrics created to 
assess writing ability abound (e.g.. Baker, Aschbacher. et al., 1992; California State 
Department of Education, 1992a; Koretz, Stecher, Klein, McCaffrey, & Deibert, 1994; Novak, 
Herman. & Gearhart, 1996; Wolf & Gearhart, 1993). Argumentation, organization, focus, 
development, mechanics, audience awareness, style, and tone are all part of good 
communication (Gearhart. Herman. Baker. & Whittaker. 1992: Wolf & Gearhart. 1993). 

As with collaboration, communication skills are content-independent; if a student can 
communicate well, he or she should be able to do so across various content areas. However, 
prior knowledge plays heavily into this ability: It is difficult to write about something you 
know nothing about. Thus, we suggest assessing communication skills within a particular 
content area. By incorporating both a concept-mapping task (assessing content understanding) 
and an explanation essay task (assessing content understanding and communication skills — see 
Baker, Aschbacher, et al., 1992) in our performance assessment package, we can evaluate 
students’ communication skills in conjunction with (and controlling for) content understanding. 
We have developed and validated a procedure for reliably scoring the essays students generate; 
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this manner of assessing content understanding implicitly includes within it an assessment of 
communication skills as well (i.e., mechanics and argumentation). That is, students who 
communicate well are better able to explain then- content understanding. 

Problem solving. A fourth type of learning involves problem-solving skills. Whether 
termed problem solving or critical thinking, it is clear that — since schools cannot possibly teach 
students everything they will need to know for the future — problem-solving skills fill the gap 
by allowing students to use what they have learned to successfully solve new problems or learn 
new skills. Industry has complained that high school graduates are unable to function well in 
the workplace because they lack the problem-solving skills necessary for success (O’Neil, in 
press). Moreover, conditions of employment are now likely to change several times during 
one’s life (Resnick, 1987; Resnick & Resnick. 1992). In such environments, it is clear that 
problem-solving skills can significantly affect an individual’s likelihood of success in the 
workforce. 

Problem solving is defined as “cognitive processing directed at achieving a goal when no 
solution method is obvious to the problem solver" (Mayer & Wittrock, 1996, p. 3). The 
CRESST model of problem solving is adapted from the problem-solving models of Glaser, 
Raghavan, and Baxter (1992) and Sugrue ( 1995). It includes four scored elements: (a) content 
understanding, (b) metacognition, (c) motivation tself-efficacy and effort), and (d) domain- 
specific problem-solving strategies. The elements are scored separately and are reported as a 
profile of problem solving. CRESST has created several feasible measures for all of these 
constructs except problem-solving strategies. To assess content understanding, CRESST has 
used both essay-based explanation tasks (e.g., Baker et al., 1995) and paper-and-pencil 
concept mapping tasks (Herl et al., 1996). We have also created measures of metacognition 
(O’Neil & Abedi, 1996) and motivation (Malpass. 1994, measuring self-efficacy; Huang, 

1996, measuring effort). In our case, the CRESST domain-specific problem-solving strategies 
are measured on a search task by looking at search behavior and the use of the information 
found. We believe that content understanding and problem-solving strategies are best assessed 
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domain-specifically whereas metacognition and motivation are best assessed as domain- 
independent constructs. However, we realize that all domain-independent constructs need to be 
instantiated in a particular domain. 

One way to assess students’ problem-solving skills is by having them search for 
information on concepts about which they are uncertain. This uncertainty principle for 
information seeking is described by Kuhlthau ( 1993) as bringing a person’s knowledge of a 
content domain from an affective state of uncertainty (a.k.a. “anomalous state of knowledge” 
or ASK; Belkin. Oddy, & Brooks. 1982) to one of understanding. Searching involves 
determining the information need; choosing topics to pursue; exploring and finding general 
information to increase overall understanding; formulating a search based on the information 
found during exploration; collecting and gathering relevant information; and. finally, presenting 
and resolving the problem (that is, finding answers or solutions to meet the initial information 
need). The search process involves the affective, cognitive, and physical realms of humans’ 
experience and is often iterative; the search queries are reformulated repeatedly as the 
information need changes (Marchionini. 1995). 

Once relevant information is found, students must integrate this newly learned material 
into their existing knowledge base. Thus, assessment of problem solving in the context of 
searching can include both evaluation of the search itself and evaluation of this knowledge 
integration. For instance, we can assess student search behavior by giving students the task of 
improving their existing representational maps (i.e.. concept maps). By monitoring their on- 
line search, we can determine whether (a) students search well enough to access rich 
information sources to support the “weak” concept areas in their maps (as determined from 
initial concept maps), and (b) students understand enough to define the problem well through 
their search behavior. This approach allows us to examine both the searching process (through 
analysis of student search strategies) and the final integrated product (the updated concept 
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map). Thus, in our technical approach, we have created both process and outcome measures of 
problem-solving strategies. 

Metacognition. The fifth and final type of cognitive learning is metacognition. 
Metacognition is defined as “knowledge or cognition that takes as its object or regulates any 
aspect of any cognitive endeavor” (Flavell. 1981, p. 37) or as knowledge about, awareness of, 
and control over one's thoughts, motivations, and feelings (Wittrock. in press). Thus, students 
who think about their thought processes, or monitor their progress, or are aware of the 
cognitive strategies they use to solve a problem are engaging in metacognitive activity. 

In our work, we characterize metacognition as consisting of four components: (a) 
awareness, (b) knowledge of cognitive strategies, (c) planning, and (d) self-monitoring or self- 
checking (O’Neil & Abedi, 1996). The second component, knowledge of cognitive strategies, 
falls into the category of metacognitive knowledge; awareness, planning, and self-checking are 
considered metacognitive activities. Metacognitive activities are assumed to be resident in 
working memory or consciousness (Dembo, 1994). 

Research on metacognitive processes suggests that students who plan and monitor their 
learning and are aware of when to use which strategies often become more active in their own 
information processing; create more complex, efficient representations; and abstract 
information better than do students who do not engage in self-monitoring activities. These 
types of results in turn lead to greater transfer (Belmont, Butterfield, & Ferretti, 1982; Berardi- 
Coletta. Buyer, Dominowski, & Rellinger, 1995; Wittrock, in press). Research has also 
shown that students can be taught metacognitive techniques which in turn can enhance their 
performance and foster transfer (Berardi-Coletta et al„ 1995; Brown, Campione, & Day, 1981; 
Lodico, Ghatala, Levin, Pressley, & Bell, 1983; Salomon, Globerson, & Guterman, 1989). 

Assessment of metacognition in empirical work can be categorized as domain-dependent 
or domain-independent (O’Neil & Abedi, 1996). Many domain-dependent studies use think- 
aloud protocols in order to elicit insights into students’ underlying thought processes (see 
Royer et al., 1993 for a review of mainly domain-dependent metacognitive assessments). 
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Domain-independent studies generally gather information about students' metacognition via 
self-report measures (see. for example, O’Neil & Abedi. 1996; O’Neil & Brown, in press; 
Pintrich & DeGroot. 1990: Weinstein. Palmer. & Schultz. 1987; Zimmerman & Martinez- 
Pons. 1990). These self-report measures have been found to be reliable with some promising 
validity information; in addition, they are clearly a much more efficient way to collect data than 
are think-aloud interviews. Including a measure of metacognition in our assessment package 
allows us not only to assess student learning of metacognitive techniques, but also to directly 
address a possible benefit of alternative assessment. 

Analyses of Innovative Technologies 
The cognitive demands analyses described below began with a large list of possible 
innovative technologies to be explored. As was discussed earlier, we are using CRESST 
assessment measures to assess government sponsored innovative educational technologies. 
Thus, we collected information on these technologies, using Web descriptions, technical 
reports (as available), information requested directly from the developers, and face-to-face 
interviews. Our decision of which innovative technologies to include for analysis was driven 
by which technologies would be in place for testing in the Fall, and by our desire to focus on 
student-level (rather than teacher-level) assessment. After reviewing all the information and 
investigating the current state of each technology, the following three technologies were 
selected for inclusion: Algebra Tutor, Hands-On Universe (HOU), and Function Machines. 

For each technology, we then requested further information from the developers, as 
available. Not only did we want access to the software itself, but we needed to gain a better 
understanding of how the software was being used. Because many of the technologies are 
relatively context-free or domain-independent (that is, they can be used in a variety different 
contexts with varied student populations), how precisely a technology is integrated into the 
classroom curriculum will affect its impact on students. Thus, we asked developers to supply 
us with specific information regarding the expected student population using their product, the 
content area in which the product was to be used, and as much curriculum material as was 
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currently available. Equipped with all this information, we then reviewed each technology or 
learning environment again, in order to analyze the types of learning afforded by each 
technology to its users. This matching of technologies with the five types of learning is shown 
in Table 1. The distinction of primary versus secondary goals was derived from the cognitive 
demands analysis. 

For each technology listed in Table 1 . the next section first includes a brief 
explanation of the technology’s purpose, intended audience, and intended use withinthe 
assessment phase. This information is essential to the analysis of these innovative technologies 
because, as explained above, many of the products are usable in a variety of ways, in a variety 
of content domains. Following these descriptions, a cognitive demands analysis is presented 
for each technology. 
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Table 1 



Matching Technologies With the Five Types of Learning: The Cognitive Demands of Five 
Technologies 



Technologies 


Content 

knowledge 


Collabo- 

ration 


Communi- 

cation 


Problem 

solving 


Metacog- 

nition 


Algebra Tutor 


XX 


X 




XX 




Hands-On 


XX 






XX 




Universe 












Function 


XX 


X 




XX 




Machines 













XX= primary goals. X= secondary goals. 



Algebra Tutor 

Purpose. The stated purpose of this technology is to “help students to develop algebraic 
skills which they can use in the context of real-life problem situations” (Anderson, Mark, 
Pelletier, et al.. 1992). This intelligent tutoring system encourages a hands-on, leaming-by- 
doing approach in which students — individually and in groups — attempt to solve real-world 
problems on-line, rather than through the use of a textbook in the classroom. Algebra Tutor 
focuses on multiple representations of information, teaching students how to use, understand, 
and interpret various types of representations (e.g., text, tables, graphs) in order to solve real- 
world algebraic problems. 

Intended audience and use. The intended audience is ninth-grade algebra students, 
both individually and in groups. Although some of the innovative technologies being reviewed 
here are relatively context-free, the Algebra Tutor is very domain-specific: This product tutors 
high school students in mathematics and algebra. Curriculum materials include problems in 
real-world mathematical applications, stressing the relevance of mathematics — and algebra in 
particular — to everyday life. 

Analysis. Students using Algebra Tutor go through a series of steps in solving each 
problem. For example, students fill in tables by identifying important aspects of the problem 
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and labeling table columns, select appropriate units of measurement, create graphs, solve 
equations, and write formulas. Algebra Tutor encourages students to become actively involved 
in the learning process. Whether used individually, in student pairs, or in collaborative groups, 
this intelligent tutor fosters learning and deep understanding of its algebra content via its 
interactive nature. Discussion (between team members), interaction (both between team 
members and between individual students and the tutor), and individualized feedback (from the 
tutor to the team or the individual student) are the key components of this system. Problem- 
solving and reasoning skills are fostered by the technology’s emphasis on representations of 
real-world problems, and — in particular — its use of word problems. 

Three types of learning are expected of students using .Algebra Tutor. First, students are 
clearly expected to gam an understanding of the algebra content presented by the program. 
Second, it is anticipated that students will improve their problem-solving skills by using this 
technology. Finally, when used in a collaborative environment, collaborative skills as well as 
enhanced understanding of the content domain should be fostered. 

Hands-On Universe 

Purposes. Hands-On Universe's primary goal is to enable high school students to 
perform genuine, real-world astronomical research in their classrooms, so as to facilitate a 
deeper understanding of astronomy and to make clear to students how mathematics and science 
are applied to real-life scientific investigations. 

Intended audience and use. The intended audience is high school science students. 

Analysis. Hands-On Universe includes four components: the HOU Telescopes (which 
capture the actual images), the HOU Telecommunications Interface (World-Wide Web pages 
used to request and retrieve images, obtain weather information, etc.), the HOU Image 
Processing Software (PC- or Macintosh-based software used for data analysis), and the HOU 
Curriculum (which includes real-world applications of math and science). Students use 
information available on the Web and in the curriculum materials to request images from one of 
several participating telescopes. They can then use the image processing software to analyze 
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their images. All four HOU components are considered research tools to facilitate scientific 
investigations, which in turn foster content understanding and problem-solving skills. 

In summary, students learn about astronomy while using the same types of tools that real 
scientists and engineers are currently using in the field. Learning expected from students using 
this technology can be found in the five families of learning. First, students are expected to 
gain a deeper understanding of the astronomy content being studied. This will include both an 
understanding of the content itself and an awareness of the importance and relevance of this 
material to real-world science. Second, by using these three technologies as research tools to 
answer pertinent scientific questions, students will enhance their problem solving skills. 
Function Machines 

Purpose. Function Machines is an application for a visual programming language 
expressly designed for mathematics and science education. This product’s intended outcomes 
are to engage students in mathematical investigations and to foster mathematical thinking and 
scientific inquiry. 

Intended audience and use. Function Machines can be used in a variety of ways, 
with a broad class of students, to study a wide range of topics. The intended audience is fifth- 
grade students; the content area to be explored is mathematics and. in particular, MathLand 
curriculum. Developers plan to have students use Function Machines in collaborative teams. 

Analysis. Function Machines uses graphical representations, which makes learning 
how to program easier and more straightforward. Using a simple metaphor of a mathematical 
function as a machine that takes something as input and produces something as output, 
Function Machines simplifies the mathematics it teaches by making everything visual. Students 
can literally see how functions work, by watching the machines in action, either step-by-step 
( step command) or in one uninterrupted sequence (“go” command). Creating machines (i.e., 
programming) in Function Machines is also visual; students construct machines using a tools 
palette, connecting machines together with drawn lines, called “pipes.” 
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Machines (or programs) in Function Machines are actually sets of simpler machines, the 
lowest level machines being predefined as primitive functions (e.g., arithmetic functions, 
graphics, logic). Students can construct machines that are either primitives or composites; 
composite machines are made up of primitives and/or other composite machines. A machine 
can also be defined in terms of itself, as can its input, which can come from any machine, 
including itself. Thus, iteration and the complex concept of recursion are both seen more 
clearly through the use of this visual programming language. The program’s beauty and 
flexibility lies in its allowing students to create anything from a very simple adding machine 
(e.g., input 1 + input2 ==> output) to a complex series (and/or encapsulated package) of 
machines to solve complex real-life mathematical problems (e.g., planning a school dance). 
During the construction and running of these programs, students can always see what is really 
going on by “x-raying” a composite machine to reveal its interior contents. 

Three types of learning are expected of students using Function Machines. First, students 
will better comprehend the specific mathematical topics (i.e., content understanding) presented 
to them while using Function Machines. Because of its visual nature, Function Machines is 
also expected to foster an inquiry-based mathematical environment in which students can 
engage in explorations and investigations. Thus, we expect students’ problem-solving skills to 
be enhanced via this software. Lastly, since Function Machines lends itself to use in 
collaborative environments (e.g., groups of students working together to solve multiple-step, 
complex mathematical problems), students using the program in this manner should gain 
collaborative skills. 

Assessment Specifications for Our Integrated Simulation 

Since we can map the types of learning we expect from students onto specific assessment 
tasks, we have assembled a suite of performance assessment tasks (our integrated simulation) 
that includes both individual and collaborative concept mapping tasks, a problem-solving 
search task, an explanation task, and a metacognitive questionnaire. The next section 
documents the domain specifications for our integrated simulation. 
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Explanation task. Using the CRESST model, students complete a short, paper-and- 
pencil prior knowledge measure, in order to activate relevant prior knowledge. Then, students 
will be presented with the paper-and-pencil explanation task. An interesting, relevant context 
will help to frame students' written responses, and the explanation prompt will require students 
to demonstrate deep understanding of the content area. These data will not be discussed today. 

Individual concept mapping task. Students are also asked to construct a concept 
map using an on-line mapping tool. The specific concepts and link labels for this task are 
provided to the students, and students work alone during this task. The individual student 
concept maps are scored on-line, by comparing them to a criterion (i.e., expert-constructed) 
map. 

Problem-solving task. Once students receive feedback on their concept maps, they 
are directed to use a constrained subset of Netscape to search for information that will justify 
improving their maps. Students are instructed to “bookmark” Web pages that they believe 
support the modifications they make to their concept maps, and to send those bookmarks to 
specific terms on the concept map. For instance, a student who finds a page with relevant new 
information about photosynthesis would bookmark the page and send that bookmark to the 
“photosynthesis” concept on his or her map. 

The information space being searched by students has been designed specifically for the 
concept mapping content area. This information space is a series of Web pages that have been 
gathered from the Internet, edited, and re-purposed for assessment use. The information space 
includes a glossary of important terms, a search engine interface, and an index. In addition, 
each page has been coded on various dimensions of relevance such that we can evaluate each 
student’s information seeking behavior as a problem-solving strategy aimed at improving an 
existing concept map. 

Collaborative concept mapping task. In this portion of the integrated assessment, 
a three-member team collaborates on constructing a concept map. One member is initially cast 
as the leader; however, each member of a group has an opportunity to assume this pivotal role. 
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The leader is the only team member that has the ability to actually change the group concept 
map; the other two members have only visual access to the map. Any changes made by the 
current leader to the group map are automatically updated to the screens of all members in a 
group. Communication between group members takes place on-line, using either predefined 
messages or an open messaging system. As in O’Neil, Chung, and Brown (in press), type of 
message usage is utilized as an index of specific group processes. The product of each group — 
a collaborative concept map — reflects the shared mental model of the group with respect to the 
relevant and important connections between and among the concepts given. Using the 
individual and group maps combined, we can gauge each student’s relative contribution to the 
group map. 

Metacognitive questionnaire. A Likert-scale trait questionnaire queried students on 
their general metacognitive activities during the integrated simulation tasks. Items targeted the 
four aspects of metacognition described above: awareness (e.g., “I am aware of my own 
thinking”), knowledge of cognitive strategies (e.g., “I select and organize relevant information 
to solve a task”), planning (e.g., “I try to determine what the task requires”), and self- 
monitoring (e.g., “I check my work while I am doing it”). 

Conclusion 

By combining all of the tasks described in the previous section, we create an integrated 
simulation based on our family of cognitive learning for students and use it for assessment of 
innovative technologies. This assessment environment is integrated both across grade levels 
and within content areas. In addition, we refer to this as an integrated assessment because it 
blends real-world, meaningful tasks with a project-based scenario that captures the many types 
of cognitive learning. The assessment is simulated in the sense that we have incorporated real- 
world activities, collaborative environments, and an Internet-like information space within a 
closed assessment. Also, the environment simulates the types of learning activities that 
innovative technologies afford in a controlled environment, which allows us to make inferences 
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regarding student learning. The following papers will provide feasibility, reliability, and some 
validity data in our approach. 
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